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ABSTRACT 


Four  government  funded  parallel  research  and  develop- 
ment projects  are  examined  to  determine  the  manner  in  which 
engineers  and  scientists  allocate  their  time,  and  the  ef- 
fect of  this  a  I  location  on  the  outcome  of  the  projects. 
The  use  of  matched  pairs  of  projects  allows  the  relative 
evaluation  of  outcomes  by  technical  monitors  in  the  cus- 
tomer agencies. 

The  percent  of  total  time  spent  in  three  categories 
of  information  gathering  (outside  consultation,  staff 
consultation,  and  literature  search)  varies  signifi- 
cantly over  the  life  of  a  project.   Higher  rated  teams 
are  relatively  stable  in  all  phases  of  information  gather- 
ing while  lower  rated  teams  initially  spend  far  more  time 
gathering  information  than  they  do  in  the  later  stages 
and  fluctuate  more  throughout  the  project.   Subsystems 
characterized  by  greater  uncertainty  receive  a  higher 
percent  of  information  gathering  time  than  subsystems 
where  uncertainty  is  lower. 


INTRODUCTION 

Research  and  development  is  a  problem  solving  process  depending  on 
information  inputs  from  the  environment  as  well  as  on  the  stored  know- 
ledge and  the  ability  of  the  personnel  involved.   improvement  in  the 
flow  of  scientific  information  is  an  important  goal  of  R  &  D  manage- 
ment, but  is  dependent  upon  a  thorough  understanding  of  the  process 
itself.   The  research  reported  here  is  concerned  with  the  manner  in 
which  individual  engineers  and  scientists  allocate  their  time  among 
three  general  information  channels  (literature;  personnel  outside  the 
laboratory;  and  personnel  within  the  lab),  and  examines  possible  effects 
this  allocation  may  have  on  the  conduct  and  performance  of  the  R  and  D 
project  on  which  they  are  working. 

To  assist  in  obtaining  a  performance  evaluation  for  each  informa- 
tion channel,  instances  were  sought  in  which  the  same  problem  is  at- 

I 
tempted  by  two  or  more  research  groups.    In  this  way,  a  comparison 

can  be  made  between  the  information  sources  leading  to  specific  solu- 
tions.  The  fairly  common  practice  of  the  Department  of  Defense  and 
the  N.A.S.A.  to  award  parallel  study  contracts  during  the  early  phases 
of  R  &  D  provides  an  opportunity  for  control  over  the  effect  of  varia- 
tions in  the  problem  substance.   The  data  presented  are  from  four  para- 
llel contracts  involving  eight  R&D  groups. 


The  idea  of  studying  parallel  R&D  projects  as  a  quas i -exper imenta I 
situation  was  suggested  by  Donald  Marquis,  who  also  helped  in  developing 
some  of  the  research  procedures. 


The  major  hypotheses  of  this  study  concern  the  variation  in  time 
allocated  among  the  three  information  channels  over  the  course  of  an 
R  i  D  project.   Specifically,  it  is  hypothesized  that: 

1.  Literature  will  be  used  more  heavily  at  the  initiation  of  a 
project  and  will  be  replaced  by  personnel  outside  the  labora- 
tory (principally  vendors)  as  the  predominant  information 
source  as  the  project  progresses. 

2.  Sources  within  the  lab  will  be  resorted  to  intermittently 
over  the  course  of  a  project. 

3.  Teams  conducting  an  extensive  information  search  at  the  begin- 
ning of  a  project  perform  better. 

4.  Subsystems  with  a  high  degree  of  uncertainty  (uncertainty  is 
operationally  defined)  receive  greater  attention  in  terms  of 
time  spent  on  information  gathering  than  do  those  which  are 
low  in  uncertainty. 

METHOD 

Once  a  para  I  lei  project  has  been  located,  its  work  statement  is  ob- 
tained and  analyzed  and  factored  into  a  reasonable  number  of  subproblem 
areas  (generally  subsystems).  The  breakdown  is  then  checked  with  the 
technical  person  who  prepared  the  work  statement,  and  data  collection 
forms  based  upon  it  are  designed.  After  all  data  have  been  collected 
from  the  contractors,  the  technical  monitor  is  revisited  and  asked  to 
provide  a  confidential  evaluation  of  each  lab's  performance.   Data  are 


gathered  by  four  means:   (I)  time  allocation  forms,  indicating  the 
amount  of  time  each  engineer  spends  on  the  job  in  several  activity 
categories;   (2)  before  and  after  interviews  with  the  individual  en- 
gineers;  (3)  periodic  tape  recorded  progress  reports  by  the  project 
manager;  and  (4)  solution  development  records. 

Time  allocation  forms  supported  by  data  from  Solution  Development 
Records  are  the  principal  source  of  the  data  here  under  consideration. 
An  example  of  the  time  allocation  forms  used  is  presented  in  Appendix 
A.   Respondents  are  asked  simply  to  record  the  breakdown  of  each  day's 
activity  among  four  categories:   time  spent  in  literature  search;  time 
spent  in  consultation  with  technical  experts  within  the  laboratory, 
but  not  assigned  to  the  project;  time  spent  in  consulting  with  experts 
outside  the  laboratory;  and  time  spent  in  analytic  design.   The  Solu- 
tion Development  Record  provides  a  means  of  following  the  progress  of 
an  R  &  D  project  through  its  subsystem  design  changes,  and  has  been 
described  in  detail  elsewhere  (Allen,  1965). 

Description  of  the  Projects 

The  four  projects  under  consideration  involved  the  following  general 
prob I  ems: 

1.  The  design  of  the  reflector  portion  of  a  rather  large  and  highly 
complex  antenna  system  for  tracking  and  communication  with  space 
vehicles  at  very  great  distances. 

2.  The  design  of  a  vehicle  and  associated  instrumentation  to  roam 
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the  lunar  surface  and  gather  descriptive  scientific  data. 

3.  The  study  of  possible  mission  profiles  for  manned  flights  to 
another  planet. 

4.  The  development  of  a  detailed  mathematical  cost  model  yielding 
complete  R  and  D  and  operational  program  cost  estimates  of 
space  launch  vehicle  systems. 

Un  it  of  Ana lysi  s 

Man-hours  devoted  to  an  information  channel  is  the  unit  of  ana- 
lysis employed,  (cf.  Menzel,  I960).   Since  engineering  man-hours  ex- 
pended is  quite  closely  correlated  with  cost  on  projects  such  as  these, 
this  choice  of  unit  provides  a  measure  not  only  of  the  relative  extent 
to  which  channels  are  used,  but  also  of  a  major  portion  of  the  relative 
expense  involved  in  such  use. 

Data  Analysis 

In  comparing  time  allocated  to  information  channels  across  pro- 
jects, the  variables  in  question  are  represented  as  percentages  of 
total  time.   This  serves  several  purposes.   First  of  all,  the  projects 
are   of  varying  size  and  using  actual  hours  to  represent  the  variables 
would  give  undue  weight  to  the  larger  projects.   Secondly,  percentage 

representation  overcomes  the  problem  of  incomplete  data  in  any  section 

2 
of  a  project.   Finally,  the  total  number  of  engineers  reporting  varies 


2 
Due  to  occasional  failure  of  engineers  to  maintain  their  time  ah 
location  forms. 


from  team  to  team  within  projects;  and  percentage  representation  allows 
intraproject  comparison. 

Percentages  for  any  given  time  period  of  a  project  are  the  ratio  of 
the  total  time  spent  by  all  individuals  on  the  variable  in  question  to 
the  total  time  reported  by  all  individuals.   Since  this  method  ignores 
the  variance  due  to  individuals  taken  separately,  all  percentages  derived 
for  the  entire  project  were  recomputed  on  an  individual  basis.   The  indi- 
vidual percentages  were  then  averaged,  and  results  compared  with  the 
aggregate  figures.   Although  this  method  gives  undue  weight  to  indivi- 
duals who  worked  only  a  small  part  of  their  time  on  the  project,  it 
does  provide  a  check  of  the  total  percentage  analysis.   Because  of  the 
inordinate  weight  given  to  part-time  participants,  individual  percen- 
tage treatment  is  used  only  as  a  check  and  mentioned  only  when  results 
differ  from  those  of  the  total  percentage  method. 

Two-sample  "t"  tests  are  used  to  test  for  differences  in  the  per- 
centage means.   Because  of  the  small  sample  sizes  employed  (four  or 
eight)  and  because  of  the  exploratory  nature  of  the  study  0.10  is 
used  as  the  cut-off  significance  level  on  a  two-tailed  test. 

RESULTS 

Relations  Between  Performance  and  Time  Allocation 

Higher  and  lower  rated  project  teams  are  determined  on  the  basis  of 
the  government  technical  monitors'  performance  ratings.   The  percent  of 
time  spent  in  total  information  gathering,  and  in  literature  search,  out- 


side  consultation,  staff  consultation,  and  analytic  design  are  averaged 
for  the  higher  and  lower  rated  teams.   The  results  in  Table  I  show  that 
on  the  average,  higher  and  lower  rated  teams  spend  approximately  the 
same  percent  of  time  in  each  of  the  five  areas. 


TABLE 


Time  Allocation  by  Function 
(Four  Paral lei  R&D  Projects) 


Higher  Lower 

Rated  Rated 


all  information  gathering  13.86^  13.43$ 

literature  search  5.02  5.32 

outside  consultation  5.09  5.32 

staff  consultation  3.77  2.00 

analytic  design  77.90  71.40 

other  8.24  15.71 


Necessity  of  Analysis  Over  Time 

Figure  I  shows  that  the  time  spent  in  gathering  information  varies 
considerably  over  a  project's  life  [f(5,I35)  =  3.32;  p<O.Or]  and  the  cc 
ponents  that  make  up  the  total  time  spent  gathering  information  vary  in  a 
somewhat  different  manner,  although  the  interaction  term  is  not  statisti- 
cally significant.   Information  gathering  is  greatest  in  the  initial  time 
period.   Literature  search  and  outside  consulting  make  up  the  largest  com- 
ponent in  this  state,  corroborating  the  general  belief  that  initially  on 
a  new  project  engineers  conduct  an  extensive  literature  search  and  seek  out 
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Figure  I.   Percent  of  total  time  which  is  spent  in 
gathering  information  as  a  function  of 

project  I i  f e  span 
(averaged  for  four  matched  pairs  of 
R&D  projects) 


possible  suppliers  for  ideas  as  well  as  components. 

While  literature  search  is  the  primary  information  gathering  tool 
in  the  initial  stages,  it  quickly  decreases  as  the  projects  progress 
and  is  replaced  in  importance  by  person-to-person  communications.   Out- 
side consulting  is  relatively  high  initially,  but  generally  speaking  is 
an  important  factor  in  all  phases  of  the  projects.   Staff  consultation 
is  the  dominant  activity  in  only  one  phase  of  the  projects,  near  the  end. 

Performance  Relations  Over  Time 

Total  Information  Gathering.   When  the  teams  are  again  divided  on 
the  basis  of  high  and  low  performance,  it  is  apparent  (Figure  2)  that 

poor  performers  spend  more  time  in  the  initial  stage  of  a  project  than 

3 
higher  rated  teams,   but  less  time  in  the  final  stage.   The  results 

are  exactly  counter  to  hypothesis  3  (page  2).   It  was  predicted  that 
those  teams  which  conduct  an  extensive  information  search  at  the  out- 
set of  a  project  would  perform  better  as  a  result.   The  results,  how- 
ever, lead  to  the  conclusion  that  the  teams  performing  the  more  ex- 
tensive information  search  were  less  well  prepared  at  the  outset,  per- 
haps through  lack  of  experience  or  failure  to  bring  themselves  up  to 
an  adequate  state  of  knowledge  prior  to  contract  initiation,  and  that 
the  more  intense  information  gathering  in  the  initial  period  fails  to 
compensate  for  this  deficiency.   As  the  project  progresses,  the  higher 
rated  teams  spend  an  almost  constant  level  of  effort  in  gathering  informa- 


3 
When  computed  on  an  individual  basis,  the  result  is  in  the  same 
direction  but  is  not  significant.   This  is  the  only  case  where  a  result 
obtained  by  the  total  percentage  method  was  not  supported  by  the  indivi- 
dual percentage  method. 
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tion,  and  end  with  a  surge  in  the  final  period.   The  lower  rated  teams, 
on  the  other  hand,  taper  off  markedly  after  the  initial  period  ar^ti   per- 
form hardly  any  information  gathering  in  the  final  period.   The  dif- 
ference in  the  final  period  may  well  reflect  the  effects  of  premature 
closure  on  the  part  of  the  poorer  performe:'S.   Since  all  of  the  projects 
were  studies  rather  than  hardware  developments,  it  may  well  be  benefi- 
cial, in  such  cases,  to  remain  receptive  to  now  ideas   right  up  to  the 
time  when  the  study  report  is  delivered  to  the  customer.    The  better 
performing  teams  apparently  behaved  in  this  manner,  actually  devoiing 
an  increased  amount  of  time  to  information  gathering  in  the  final  per- 
i  od . 

Outside  Sources.   Higher  rated  teams  are   far  more  consistent  over 
tim.e  than  lower  rated  teams  in  their  use  c^  information  sources  out- 
side the  laboratory.   The  higher  rated  teans  seem  to  stay  closer  to  the 
ideal  of  bringing  outsiders  "on  board'  early  and  maintaining  close  con- 
tact throughout  the  project's  life,  not  just  when  difficulties  arise. 
In  this  way  both  parties  are  kept  sufficiently  aware  of  progress  made 
by  the  other,  and  such  difficulties  as  interface  incompatibility  are 
reduced.   The  lower  rated  teams,  on  the  other  hand,  do  not  maintain  close 
liaison  with  the  external  sources,  and  periodic  spurts  of  heavy  communi- 
cation fail  +o  compensate  for  the  lack  of  a  continuous  relationship. 

Technical  Staff.   Although  the  time  spent  with  this  channel  is  the 
smallest  reported  (2.0^  for  losing  teams  and  3.8^  for  higher  rated  teams), 
it  is  a  very  important  source  of  technical  information.   Staff  specialists 
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are  often  ca I  led  upon  to  solve  critical  problems  beyond  the  capa- 
b i li ty  of  team  members.   Again  the  time  pattern  follows  that  estab- 
lished for  outside  sources  --  consistency  over  time  by  the  higher 
rated  teams  and  fluctuation  by  the  lower  rated  teams.   Higher  rated 
teams  spend  significantly  more  time  only  in  the  last  stage  of  pro- 
jects (Figure  4). 

Literature  Search.   Although  the  differences  between  higher  and 
lower  rated  teams  are  not  significant,  the  general  pattern  is  simi- 
lar to  that  for  the  other  information  channels.   Higher  rated  teams 
remain  fairly  steady  and  lower  rated  teams  fluctuate.   The  lower  rated 
teams,  however,  are  not  quite  as  erratic  as  in  other  information  acti- 
vities, showing  instead  a  rather  steady  decline  as  the  project  pro- 
gresses (Figure  5). 

Uncertain  Subsystems 

When  there  is  great  uncertainty  about  the  outcome  of  the  solution 
approaches  under  consideration,  engineers  will  either  seek  out  new  ap- 
proaches or  try  to  reduce  the  uncertainty  in  the  old  ones  through  at- 
tainment of  greater  knowledge.   In  either  case  information  is  gathered 
to  aid  their  decision.   Therefore,  it  was  hypothesized  that  subsystems 
characterized  by  high  uncertainty  would  require  more  information  gather- 
ing than  those  in  which  uncertainty  is  low. 

To  fest  this,  it  is  first  necessary  to  define  and  measure  uncertainty, 
At  least  two  estimates  of  uncertainty  are  available  from  the  data.   From 
the  Solution  Development  Record,  the  average  weekly  value  for  Lp   can  be 
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Figure  4.   Comparison  of  percent  time  spent  by  higher 

and  lower  rated  project  teams  in  consultation 
with  the  lab's  technical  staff 
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4 
determined  for  each  subsystem .   A  large  value  of  Ap  should  indicate  high 

uncertainty  but  in  some  cases  several  alternatives  are  carried  at  a 
low  level  throughout  most  of  the  project  with  none  achieving  clear 
dominance;  in  such  cases  Ap  can  be  very  small  and  yet  the  uncertainty 
is  obviously  high.   A  more  reliable  and  sensitive  measure  appears  to 
be  the  number  of  weeks  the  winning  alternative  maintains  a  probability 
less  than  0.7  on  the  Solution  Development  Record.   If  the  winning 
alternative  is  reported  at  a  probability  of  0.7  or  higher  throughout 
most  of  the  project,  the  subsystem  is  one  with  a  relatively  high  de- 
gree of  certainty.   Conversely  if  the  probability  of  acceptance  ex- 
ceeds 0.7  for  only  a  small  portion  of  the  project,  this  would  indicate 
uncertainty  relative  to  the  design  of  the  subsystem. 

Uncertain  subsystems  are  thus  defined  in  the  following  two  ways: 
(I)   Those  whose  final  solution  maintained  a  0.7  or  higher  probabi- 
lity for  a  number  of  weeks  which  was  less  than  the  average  for  the  pro- 
ject.  (2)   Those  with  a  weekly  Ap  greater  than  the  average  for  the 
project . 

The  two  operations  research  projects  are  omitted  from  this  ana- 
lysis because  in  these  instances  subsystems  are  not  clearly  defined, 
and  relative  uncertainty  is  therefore  impossible  to  determine.   In  each 


4 
Engineers  provide  a  weekly  estimate,  on  the  Solution  Development 
Records,  of  the  subjective  probability  that  each  alternative  considered 
as  a  possible  solution  will  in  fact  be  finally  chosen  as  the  solution. 
The  measureAp  is  the  total  change  in  probability  level  for  all  alter- 
natives in  a  given  subproblem  (Allen,  1955). 
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of  the  other  two  projects,  it  is  possible  to  identify  twenty-one 
individual  subsystem  designs.   There  is  some  overlap  among  indivi- 
duals in  their  responsibility  for  subsystems;  in  these  cases  assign- 
ment is  made  equally  to  each.   This  is  a  conservative  procedure  which 
results  in  a  general  weakening  of  correlations  for  the  subsystems 
affected . 

With  these  limitations  noted,  the  percent  of  information  gathering 
devoted  to  each  subsystem  is  compared  with  the  project  average.   Figure 
6  shows  the  results  of  Fisher  exact  probability  tests  (Siegel,  1956) 
of  the  hypothesis  for  each  of  the  definitions  of  an  uncertain  subsystem. 

These  results  indicate  that  the  engineers  working  on  problems  of 
higher  uncertainty  do  allocate  more  time  to  information  gathering  than 
on  problems  of  lower  uncertainty.   This  fact  should  be  recognized, 
and  special  provision  made  to  supply  those  engineers  who  are  working 
on  subsystems  which  are  relatively  complex  or  are  pushing  the  state- 
of-the-art  with  the  necessary  technical  information  for  their  work. 
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Certainty  based  upon  number  of  weeks  in  which  final  solution 
exceeded  0.7  probability  of  acceptance,* 


Percent  of 
Total  Time 
Spent  in 
I nf ormat  ion 
Gather! ng 


uncerta  i  n 
subsystems 


High 


Low 


certa  i  n 
subsystems 


10 

5 

1 

4 

probab  i  lity  of 
occurrence  under 
the  nu I  I  hypothe- 
sis  =  0.097 


*N  reduced  by  one  as  a  result  of  discarding  one  subsystem 
which  fell  on  the  median  of  the  uncertainty  measure. 


Certainty  based  upon  weekly  probability  change. 


Percent  of 
Total  Time 
Spent  in 
I nf ormat  i  on 
Gather  i  ng 


uncerta  i  n 
subsystems 


High 


certa  i  n 
subsystems 


Low 


5 
4 


probabi  I i ty  of 
occurrence  under 
the  nu I  I  hypothe- 
sis  =  0.097 


Figure  6.   Proportion  of  time  devoted  to  information  gathering  as 
a  function  of  subsystem  certainty. 
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CONCLUSIONS 

Four  pairs  of  government  funded  parallel  research  and  development 
projects  are  examined  to  determine  how  their  engineers  and  scientists 
allocate  their  time,  and  the  effect  of  this  allocation  on  the  outcome 
of  the  projects.  The  use  of  matched  pairs  of  projects  allows  the  rela- 
tive evaluation  of  outcomes  by  technical  monitors  in  the  customer  agencies. 

Three  relationships  are  investigated:   the  percent  time  spent  in 
three  information  gathering  activities  as  a  function  of  the  phase  of 
the  project;  the  relation  of  time  allocation  patterns  to  performance; 
and  the  correlation  between  degree  of  uncertainty  in  a  subsystem  and 
the  amount  of  information  gathering  devoted  to  that  subsystem. 

The  percent  of  total  time  spent  in  all  phases  of  information 
gathering  varies  significantly  over  the  duration  of  projects.   When  the 
activities  are  measured  over  time, a  clear  relation  with  performance 
emerges.   Higher  rated  teams  are  more  consistent  over  the  duration 
of  the  projects  in  all  phases  of  information  gathering,  while  lower 
rated  teams  initially  spend  more  time  gathering  information  than  they 
do  in  the  later  stages  and  fluctuate  more  throughout.   Literature  search 
by  lower  rated  teams  is  regular  in  its  declining  use  from  the  beginning 
to  the  end  of  projects.   These  results  hold  for  two  methods  of  computing 
percent  time  spent  in  any  activity. 

The  time  of  R  i  D  engineers  is  a  most  valuable  commodity.   Much  of 
this  time  is  spent  in  seeking  and  gathering  needed  technical  information. 
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Certain  procedures  should  be  designed  to  increase  the  efficiency  of  this 
search  in  all  instances,  but  where  a  choice  must  be  made,  steps  should 
be  first  taken  in  areas  where  the  problem  is  most  critical.   The  present 
research  indicates  that  one  mechanism  for  determining  such  critical ity 
is  an  estimate  of  the  relative  uncertainty  of  solution  of  each  subproblem. 
Since  more  uncertain  subproblems  are  in  greater  need  of  technical  informa- 
tion, the  engineers  assigned  to  such  subproblems  should  be  provided  spe- 
cial assistance  in  gathering  needed  information. 
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APPENDIX 

Time  Allocation  Forms  are  specially  designed  time  sheets  which 
permit  the  engineer  to  indicate  the  manner  in  which  he  divides  his 
time  among  four  modes  of  information  gathering  and  processing  activity, 


MASSACHUSETTS  INSTITUTE  OF  TECHNOLOGY 
Time  Allocation  Form 


Name 


Date 


^"^"^^^^^  Day 
Activity    ^"^^>^^^ 

Sun . 

Mon. 
DATE 

Tues. 

Wed. 

Thurs. 

Fri. 

Sat. 

Analytic  Design 

1 

Literature  Search 

Consultation  with  Spec- 
ialists within  the  Com- 
pany (but  not  on  the 
research  team) 

1 
i 

Consultation  with  Experts 
outside  of  the  Company 
(paid  or  unpaid,  formal 
or  i  nforma 1 ) 

I 

Total  time  on  the 
Prob 1  em 
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